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OBJECTIVE: 

We  strive  to  improve  human  performance  through  the  adaptive  filtering  of  information. 

APPROACH: 

We  conducted  a  series  of  empirical  studies  to  determine  what  factors  impact  human  performance  when  users 
extract  and  use  information  obtained  from  complex  computer  displays.  In  order  to  understand  how  users 
extract  relevant  information,  we  first  needed  to  understand  the  nature  of  complexity  in  computer  displays. 
Our  initial  studies  showed  that  complexity  in  computer  displays  is  a  function  of  both  the  task  and  the 
individual  What  may  be  easy  for  one  person  is  hard  for  another  and  what  is  easy  at  one  time  for  a  person 
may  be  hard  for  the  same  person  at  another  time.  If  we  can  determine  the  complexity  of  a  computer 
display  for  a  user  at  any  particular  time,  then  we  can  instantaneously  adapt  the  display  to  optimize 
information  throughput  for  this  user’s  current  capabilities.  To  do  this,  we  must  provide  the  computer  with 
information  about  the  state  of  the  user.  Our  measurements  of  the  user  are  done  with  an  army  of  different 
physiological  sensors  as  different  sensors  give  different  information  for  different  tasks.  Understanding  how 
users  extract  information  is  facilitated  by  eye-movement  studies.  The  type  of  tasks  we  investigated 
included  searching  for  information  [b2,  b6,  b8,  plO],  making  decisions  (information  veracity  and  finding 
optimal  routes)  1J5,  jl  1,  b7,  b8,  bl2,  p2,  p5,  p8,  plO,  pl4,  pl5,  pl6J,  understanding  text  (especially 
native  vs.  second  language  readers)  03,  j6,  J8,  j7,  j9, )  13,  p26]  and  comprehending  computer  programs 
[b2,  pl3].  Information  on  the  computer  displays  that  we  studied  included  maps  (icons  and  routes)  [blO], 
procedural  instructions,  computer  programs,  games  [p4,  p20],  web  sites  [bl,  p2,  p7,  plO,  pl4,  pig,  p!9* 
p32,  p38],  and  judgments  of  proportions  (fractions)  with  different  visualizations  [b3,  b9,  p22,  p31]. 

ACCOMPLISHMENTS  ( entire  grant  periods 

We  built  various  instrumented  experimental  software  test-beds,  including  the  EMI  (Evaluating  Multi-user 
Interfaces)  crisis-management  test-bed  [bl  1]  and  the  MTF  (Moving  Target  Fractions)  reeonfigurable  test¬ 
bed  [bl3,  pi  7],  which  is  capable  of  being  reconfigured  to  handle  any  number  of  event-driven  inputs  (either 
in  the  test-bed  or  via  serial  ports  or  over  TCP/IP  networks)  simply  by  editing  a  text  file  (in  XML  format). 
Even  the  testbed  application  can  be  easily  changed.  Using  these  test-beds,  we  performed  experiments  with 
human  participants  that  found  filtering  of  background  complexity  in  maps  during  optimal  route  finding 
was  helpful  overall.  However,  we  found  considerable  individual  differences  in  both  the  users’  search 
strategies  and  their  perception  of  task  complexity  011,  j  12,  b6,  bl3].  The  strong  individual  differences 
inspired  us  to  look  for  ways  to  measure  these  differences  and  to  break  the  problem  down  into  its 
components.  This  strategy  helped  us  understand  the  complexity  of  the  basic  building  blocks  before 
recombining  them  into  larger  problems.  The  basic  building  blocks  we  studied  included  text  (labels  in 
maps),  graphs  (road  networks),  and  icons. 

To  provide  the  computer  information  about  individuals  in  a  continuous  unobtrusive  manner  so  that  the 
computer  can  adapt  the  presentation  of  Information  to  the  individual  user,  we  designed  new  physiological 
sensors,  including  a  Pressure  Mouse  (a  regular  computer  mouse  instrumented  with  pressure  transducers  on 
the  body  and  buttons)  and  finger  clips  to  measure  galvanic  skin  response  (GSR),  blood  flow  and 
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skin/ambient  temperatures  and  associated  electronics  for  input  of  the  measurements  into  a  computer  as 
shown  in  Figure  1  [b4,  p4,  p9,  pl2,  p20],  Using  the  test-bed  software,  we  controlled  for  the  cognitive 
difficulty  of  tasks  and  found  correlations  in  the  physiological  measures.  We  combined  these  biosensor 
measurements  algorithmically  in  real-time  into  a  number  of  “gauges”  that  represent  different  aspects  of  the 
user’s  state  including  arousal  &  stress,  perceptual  &  motor  load,  and  cognitive  difficulty.  These 
biosensors  allow  us  to  measure  differences  among  people  (e.g.,  the  Pressure  Mouse  can  distinguish 
between  people  in  2-3  mouse  clicks)  as  well  as  differences  in  a  single  person  as  their  cognitive  state 
changes  over  time. 


We  investigated  the  basic  building  blocks  of  computer  displays  by  conducting  empirical  studies  of  text, 
specifically  finding  and  judging  the  veracity  of  information,  understanding  procedural  instructions  and 
understanding  simple  computer  programs  [pi 3],  We  found  that  even  non-experts  could  reliably  judge  the 
veracity  of  domain  information  on  the  Web  if  the  non-experts  had  had  some  instruction  in  general  search 
and  veracity  judgment  skills  [plO],  For  understanding  procedural  instructions  and  computer  programs,  we 
found  the  format  (especially  indenting  following  the  logical  nesting  of  the  instructions/programs)  was 
significant  in  influencing  understanding.  We  also  found  that  we  could  recognize  experts  versus  novices 
simply  from  noting  that  experts  fixated  longer  on  the  most  salient  text.  We  could  also  distinguish  native 
versus  second-language  readers  based  on  both  the  reader’s  scan  patterns  and  the  mouse  usage  patterns.  We 
found  added  cognitive  load  from  split  attention  when  text  and  associated  graphics  were  placed  even  a  little 
bit  apart.  We  also  found  that  most  readers  had  trouble  comprehending  graphics  without  meaningful  labels 
and  some  readers  did  not  even  fixate  on  any  graphics  at  all. 

We  studied  the  difficulty  of  finding  and  enumerating  multiple  arrow  targets  (similar  to  airplane  depictions 
in  radar  screens)  in  static  displays  with  varying  difficulty  of  backgrounds  (no  distractors,  obliquely  oriented 
arrows  and  the  more  difficult  oppositely  oriented  arrows)  as  shown  in  Figure  2  [b6].  The  inspection 
patterns  of  fixations  supported  the  idea  that  pre-attentive  processes  play  a  substantial  role  when  targets  are 
few  in  number,  easy  to  locate  and  easy  to  discriminate  from  each  other.  Our  results  provide  evidence  for 
the  operation  of  holistic  enumeration  processes  that  work  pre-attentively  at  least  in  arrays  with  no 
distractors.  When  we  added  color  to  orientation  of  distractors  using  elongated  ovals  (similar  to  the 
depiction  of  ships),  we  found  similar  results.  We  next  studied  the  influence  of  visual  layout  by  performing 
empirical  studies  of  enumerating  nested  versus  adjacent  squares  and  found  that  adjacent  squares  were 
significantly  easier  to  process. 
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e  studied  the  perception  of  proportions  (essential  for  gestalt  understanding  of  maps)  by  empirical  studies 
ol  proportion  comparison  judgments  using  various  representations  of  proportions  including  two  pairs  of 
mes  (either  parallel  or  perpendicular  pairs)  or  two  rectangles.  The  rectangles  were  far  easier  to  compare 
than  perpendicular  lines,  which  m  turn  were  easier  than  parallel  lines.  We  repeated  the  experiment  with 
different  shapes  like  ovals  and  circles.  We  found  only  slight  differences.  Very  importantly,  we  found 
significant  correlations  between  mean  fixation  durations  in  each  area  and  complexity  (both  number  of 
targets  and  background  complexity)  in  all  enumeration  and  proportion  judgment  tasks.  This  is  a  new 
easy-to-caiculate  measure  of  cognitive  load  in  these  types  of  visual  search  tasks. 

In  order  to  test  our  biosensors,  we  wanted  to  add  stress  to  the  perception.  Using  a  game  of  preventing  the 
sprea  of  an  infectious  disease  (this  was  long  before  SARS),  the  user  has  to  click  (representing  treatment) 
on  a  square  before  some  time  limit  from  the  start  of  infection  in  that  square  to  prevent  the  spread  of  disease 
o  t  e  neighboring  squares.  Over  time,  increasing  numbers  of  random  squares  become  infected.  The  user 
was  given  mformation  about  the  progress  of  disease  in  each  square  by  either  changing  color  from  green  to 
red  as  m  Figure  3,  giving  a  changing  ratio  (numeric  fraction),  or  filling  in  a  square.  We  found  the  filling 
square  more  effective  th^^^angmgcolOTj^ichwas  much  more  effective  than  the  ratios  [b3,  p20]. 


Figure  3.  Virus  contamination  task. 

To  determine  whether  cognitive  load  can  be  detected  with  our  new  biosensors,  we  designed  a  game  where 

nv!u  rtf?*0  ?  ‘Ck  °n  only  thos.e  fractions  greater  than  1/3.  As  shown  in  Figure  4,  the  fractions  are  in 
ovals  that  float  m  various  directions  across  the  screen.  Correct  selections  turn  green  and  increase  the  user’s 
score  while  incoirect  selections  or  missed  selections  (that  leave  the  screen)  turn  red  and  decrease  the  user’s 
f^e;/1C°gn,J/ve  !oadcan  be  controlled  by  changing  the  difficulty  of  fractions  (e.g.,  3/4  is  easier  to  judge 
than  6/17)  and/or  by  changing  the  rate  of  new  fractions  and/or  the  speed  of  fractions  (all  changeable  by 
editing  text  parameters  m  MTF’s  XML  file).  We  found  correlations  between  the  physiological  i 
2nd  obvious  workload  me  asures  of  the  task  2$  shown  in  F12  ^ 


l  measures 


Figure  4.  MTF  testbed  with  fractions  floating  across  the  screen  and  the  score  on  the  bottom. 
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Figure  5.  Correlation  of  biosensors  with  obvious  task  workload  measures. 


After  detecting  cognitive  workload  in  real-time,  various  user  interface  adaptation  techniques  can  be  applied 

such  as  information  filtering  [b2,  b4]  or  adapting  the  presentation  of  information  to  the  user’s  cognitive 

load  instead  of  or  in  addition  to  other  user  characteristics  [j5,  j 6,  j8,  b5,  bl  1]. 

SIGNIFICANCE: 

The  research  supported  by  this  project  has  produced  results  that  will  help  guide  the  design  of  future  crisis 

management  and  battlefield  management  interfaces.  In  particular,  our  completed  experiments  show: 

1)  Eye  fixation  durations  are  strongly  correlated  with  both  background  complexity  and  number  of  targets 
in  a  simple  visual  search  task.  So  eye  fixation  durations,  which  are  very  easy  to  calculate,  can  be  used 
as  a  measure  of  cognitive  load  in  simple  visual  search  task  and  possibly  in  other  types  of  tasks. 

2)  Hand  and  finger  pressures  on  a  mouse  are  correlated  with  cognitive  load  in  target  selection  tasks.  So 
hand  and  finger  pressures  on  a  mouse,  which  can  be  collected  in  a  completely  unintrusive  manner,  can 
be  used  as  a  measure  of  cognitive  load  in  simple  target  selection  tasks. 

3)  Finger  pressures  during  a  mouse  dick  are  highly  individual.  Within  three  to  five  clicks,  identification 
of  the  individual  using  the  mouse  can  be  established  to  90%  accuracy.  So  click  signatures  can  be  used 
to  help  establish  the  identity  of  the  user  much  like  other  biometric  identification  technologies.  The 
advantage  of  this  biometric  is  that  it  is  much  cheaper  than  other  biometrics  like  fmgerprint/handprint 
or  retinal  scanners,  although  it  is  also  less  accurate  than  these  alternatives. 

4)  Nested  objects  are  significantly  more  difficult  to  count  than  adjacent  objects. 

5)  Performance  significantly  decreases  with  both  map  complexity  and  task  complexity. 

6)  Up  to  three  static  distractors  make  no  difference  in  map  route  finding  performance. 

7)  Increasing  numbers  of  moving  distractors  correlates  with  decreasing  map  route  finding  performance. 

8)  Having  one  distractor  move  at  a  time  makes  no  difference  in  map  route  finding  performance. 

9)  Judgments  of  ratios  (represented  by  line  length)  are  significantly  easier  when  the  lines  are  presented 
two-dimensionally  (as  rectangles)  rather  than  linearly. 

10)  Training  in  one  graphic  data  model  style  does  not  transfer  readily  to  other  graphic  styles  even  for 
related  models. 
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